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ABSTRACT 

The emphasis of mastery learning instruction is 
unique in that concern is not only with raising the level of 
achievement but also with reducing the variability of achievement 
among the- students instructed. The Glass effect size estimator, 
presently used an an index to measure the effectiveness of mastery 
learning experiments, is examined relative to these two conditions. 
This paper advances the thesis that the Glass effect size estimator 
represents only one dimension in the assessment of mastery learning 
experimental outcomes. The second dimension of effect, relevant to 
mastery learning outcomes, is the variability of the effect. The 
index to be proposed is directed at assessing this condition, in 
addition to the information provided by the effect size estimator. 
(Author) 
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ABSTRACT 

The emphasis of mastery learning instruction is unique in that 
concern is. not only with raising the level of achievement but also 
with reducing the variability of achievement, among the students 
instructed. The Glass effect size estimator, presently used as an 
index to measure the effectiveness of mastery learning experiments, 
is examined relative to these two conditions. This paper advances 
the thesis that the Glass effect size estimator represents only 
one dimension in the^assessment of mastery learning experimental 
outcomes. The second dimension of effect, relevant to mastery 
learning outcomes, is the variability of th€"^ff6ct. The index to 
be proposed is directed at assessing this condition, in addition 
to the information provided by the effect size estimator. 



A PROPOSED ADDITIONAL INDEX TO GLASS" EFFECT SIZE ESTIMATOR 
WITH APPLICATION TO MASTERY LEARNING EXPERIMENTS 

Stimulated by the publication of Jacob Cohen's book , Statistical 
Power Analysis for theBehavioral Sciences (1969), and popularized 
by Glass and others (Glass, 1976; Smith, and Glass, 1977; Rosenthal 
and Rubin, 1978; Kulik, Kulik and Cohen, 1979; Glass and Smith, 
1979), the concept of effect size (ES) has received much attention as 
a practical means of assessing treatment effectiveness and as a tool, 
for the quantitative synthesis of research. Of particular interest 
in this paper is the utilization of the Glass effect size estimator 
as an index to measure the effectiveness of individual mastery 
"learning (ML) experiments. The emphasis of mastery instruction is 
unique in that the ML philosophy is concerned not only with raising 
the level of achievement but also with reducing the variabil i ty of 
achievement. "As we have proposed elsewhere (Block, 1974), if 
mastery approaches do in fact' help more students to learn better 
than has traditionally been the case, the mastery-taught students 
should exhibit greater learning, as well as less variability in 
their learning, than nonmastery-taught students. That is, the 
mastery-taught students should not only learn better, / they should 
learn more like one another 11 (Block and Burns, 1977). 



Traditional instructional . model s are not specifically concerned, 
a priori , with the dual problems of increasing the level of achieve- 
ment and concomitantly reducing the variability of performance as the 
major outcomes of instruction. In these models, concern is usually 
concentrated upon the overall level of treatment effectiveness (effect 
size) as measured by the differences in average level of achievement. 

Benjamin Bloom believes that the outcome of reduced variance 
resulting from ML experiments is one useful index of what his 
theory of school learning promises (Bloom, 1976). Block and Burns 
(1977) in their review of 39 ML experiments involving 97 compari- 
sons of average achievement test scores in a variety of subject 
areas report that mastery-taught students scored higher than 
nonmastery-taught students 89 percent of the time, and significantly 
higher 61 percent of the time. In addition, 26 of the 39 reviewed 
ML experiments reported variance data involving 80 comparisons. Of 
the 80 comparisons, mastery students exhibited less variability 74 
percent of the .time on achievement' test scores>than nonmastery- 
taught students. Upon computing the ES estimator for each of these 
comparisons, Block and BurnSx report that mastery-taught students, 
scored on the average, two-thirds of a standard deviation higher 
than nonmastery-taught. students. Similar results are reported by 
Bloom (1976). 
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It is clear then that the dual importance accorded to level and 
variability of achievement within the ML context necessitates the 
development of indices reflecting these conditions. The measure- 
ment of levels of achievement, vis a vis effect size, is appropri- 
ate in "traditional " instructional experiments as. noted previously. 
However, this measure represents only one dimension in the assess- 
ment of ML experimental outcomes. The second dimension of effect, 
relevant to ML experiments, is the "variability" of the . effect. 
The index to be proposed is directed at assessing this condition in 
addition to the information provided by the ES estimator. 

THE GLASS EFFECT SIZE ESTIMATOR 



The ES estimator as proposed by Glass (1976) is defined as the 
difference between the means of the experimental (Yg) and control 
groups (Tq) divided by the standard deviation of the control group 
(Sq). The index is a pure, "dimensionl ess," number expressed in 
terms of the standard deviation metric. Simply stated, M . T . the 
effect size (ES) is some specific non-zero value in the population. 
The larger this value, the greater the degree to which the phenom- 
enon under study is manifested" (Cohen, 1977)r Thus, ES = 1.0, 
indicates that the experimental group mean exceeds the control 
group mean by one standard deviation; assuming normality, only 16 




percent of the control group participants are higher than the 
average experimental participant. This' interpretation is justified 
if the two groups compared are distributed normally on the outcome 
measure and have homogeneous variances. However, these assumptions 
are not tenable for an ML experiment where the conditions of non- 
homogeneous variances and non-normality are likely to result, (In 
a latter section of this paper, pertaining to the application of 
the proposed index to the results of actual ML experiments, a test 
of the non-homogeneity of variances is conducted to verify the 
outcome of this condition..), j 



Although Glass (1978) notes that the definition of ES appears 
uncomplicated, he acknowledges that heterogeneous variances pose 
substantial difficulties in its formulation and interpretation. 
He then proceeds to present his arguments for utilizing the control 
group standard deviation as the metric of standardization- via the 
consideration of the following examples: 



Experimental ■ 



Control 



Means 



X E = 52 



X C - .50 
S C ■ 10. 



Standard Deviations 



S E = 2. 



Basis of Standardization 



Effect Size 




1.00 



(S E + S c )/2 



0.20 



0.33 
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He states that the measure of ES could be calculated by using 
either S£, Sc or a combination of both, but notes the huge 
differences resulting. The average of the two standard deviations 
should be dismissed in his words "as merely a mindless statistical 
reaction to a perplexing choice" (Glass, 1978), and argues that the 
two remaining estimates are correct. \ 



In Glass' second example, he considers two experimental groups 
compared with a control with the followinq results: 



Me 

Means 

/ 



:hod A Method B Control 
50 50 48 



Standard Deviations 10 1 4 

* 

His argument follows: 

If effect sizes are calculated using the standard 
deviation of the "method, 11 then ES A equals 0.20 and 
ESb equals 2.00 a misleading difference, con- 
sidering the equality of the method means on the 
dependent variable. Standardization of mean 
differences by the control group standards devia- 
tion at least has the advantage of al lotting. equal 
effect sizes to equal means. This seems reason 
enough to resolve the choice in favor of the control 
group standard deviation (Glass, 1978).. 



er|c io 



He further notes, however, that the problem of heterogeneous 
variances remains and has no clear answer. 



In the preceding example, if the mean of the control group 
happened to equal 50, it is clear that the ES estimator would equal 
zero in both instances, implying no differences between the groups. 
This is a perplexing problem for measuring the effectiveness of 
mastery learning outcomes. If Method B represented a ML group, 
intuition would suggest a difference favoring that condition, i.e., 
although the level of learning is the same for all three groups, 
those individuals in Method B^are learning, as a group, in a more 

homogeneous manner than those l in Method A and the Control 1 The 

' / ■ \ 
problem here'is one of accounting for the presence of heterogeneous 

variances, when Glass 1 estimate of effect size is zero J 

■ \ \ ■ 

A PROPOSED SOLUTION 

The effect variability index has the property thatj it utilizes 
all the descriptive statistics resulting from an experiment, i.e., 
the means and standard deviations of the groups to be compared, 
and does not go to zero in the presence of equal means associated 

with heterogeneous or homogeneous variances. Its utility will be 

■ \* 

\ 



Illustrated by comparing ft against the results of the two previ- 
ously cited examples, 1n addition to applying It to actual ML 

experimental results to Illustrate Its Interpretive power and 

i 

ease of appl 1cat1on. 

The proposed Index is derived by creating for each experimental 
group 1 the coefficient of variation, i.e., the ratio of the sample 
standard deviation to the sample mean (Snedecor and Cochran, 1980), 
The i nclex is then formed by etablishing the ratios of the coefficients 
of variation for the groups ^ »>e compared. That 1s, 

Ii/j - CVi/CVj j i = j i = j = l...k treatment (1) 

_ _ groups 

= (si/*i)/(sj/Xj) 

= (si/SjHXj/Ii) 

? 

l 

The impetus for this formulation is that "each coefficient of vari- 
ation reflects the amount of variability associated with a given 
mean, and subsequently, the ratio of^these coef ficients^fadl 1 tates 
the asses' ment of their combined effectiveness^ The interpretation 
of this index can best be illustrated by referring to Glass 1 first 
example: 




Experimental Control 
Means ; "Xe = 52 . 7c = 50 

Standard Deviations Sr = 2 Sr =10 . 

I ' •■ . ■ 

I Coefficients of Variation 2/52 = .038 10/50 = .20 

: ' & 

! ■ 

I ES (using control group SD) 0.20 

Recalling the. definition of I i / j » ' 

' ^ IE/C = (CVe/CVc) - , 

IE/C = (Se/S C )-(X C /Xe) 

and substituting , 

l£/C = (2/10) -(50/52) 

• . IE/C =. (.2)-(-96) » 

IE/C = 49 ; i 

' ' \ ■ ■ 

The importance of this index. is not necessarily Reflected in 
■ ; ■ ; \ . 

the final number, but in the information provided by -considering V 

the factors determining the product. Each factor presents the- 

relative impact of the individual ratio of the means and standard 

deviations. Let us explore this result in the context of an ML 

experiment. \ 

• \ ' ' ■ ■ y" 

: The ES estimator equals 0.20, suggesting that the two groups 

.do not differ greatly . Al though^^is estimate may result in a 

satistical 1y, significant difference in t fve -traditional sense, the 

magnitude of the number is considered.a "small" efTec-t^wi thin the 



xontext of ML experiments (Block and Burns, 1977; Bloom, 1976) • 
In spite of the levels of achievement being relatively similar, 
the argument coul d be, advanced that the lower achieving ML (Experi- 
mental) students are for the most part all achieving at a . high 
level compared to the lower achieving students in the control group 

(S E = 2 vs S C = 10). 

/ / ■ i 



Althougt/the two indices appear similar, ES = ,20 and Ie/c 
= .19, their meanings are different. The closer Ie/C is to zer0 
the greater the impaction variability of the experimental treats 
ment; (i.e., the group represented in the numerator); whereas^ the 
larger the value of ES (either in a positive or negative direction) 
the greater the effect Vavoring either the experimental or control 
group. As Ig/c approaches one , the di fference between the two 

groups becomes minimal . As I^/c exceeds one i the effect variability 

. * ' * 1 v . 
tends to favor the control group (i .e. , the group represented in 

the denominator). Finally, where the ES estimator disappears in 

the presence of equal experimental means and heterogeneous or 

homogeneous variances, the proposed estimator retains thi s information 

the ratio of the means is- simply on^ and the index is soley dependent 

. / . s " 
on the ratio. of standard deviations. 

Let us illustrate this last situation by assuming that the 
colvtrol group mean in the present, example equals 52. The ES 
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estimator equals zero (no difference in experimental effect) 
whereas. I E / C = nonessential ly equivalent to I E/C = .19, in which 
T E = 52 and Y c = 50; a result primarily due to the approximate 
equality of the means compared, i.e., T C /T E = 50/52 = .96, whereas 
Xq/X e = 52/52 = 1.0. Thus, the effect of heterogeneous variances 
is clear in the presence of equal or nearly equal means. , 

It is noted, thai: the formulation of this ratio does not neces- 
sitate that the coefficient of variation of the experimental group 
appear in the numerator. For example, we could use then, 

ic/E = (s c /Se)-(Xe/Xc) (2f 

and substituting yields, 

i ; 

IC/E = (10/2). (52/50) 
k/,£ = (5) • (1^04) , 
- IC/E = 5.2 = 1/lE/C 

■i : ■ - ' 

Since the index exceeds one, the results favor the experimental 
group, i.e., the group represented in the denominator. Thus, \£/q 
and I C/E are reciprocally related. This relationship together with 
a comparison of I E / C and ES are summarized in Table 1. As is tri/e 
for all newly proposed quantitative indices, terminology such as 
"close to zero" or "close to one" carry a subjective interpretation, 
eventually refined by familiarity with having employed the index in 



TABLE 1. Characteristics of l E /c 



Its 



Equl vaJent 
I nterpretatlon Cond I tlon 



Cond It Ions 



Interpretations 



'E/C 



ES 



approaches 
zero 



exper Imental 
group favored 



'C/E approaches j' 
one 



I . . S" E > S c & Xc > x E 



• 75 L 'E/C ± 1 - 0 ' 
no d I f ference 



sma I I & negative , 
no dl f ference 



approaches 
one 



no dl f ference 
between groups 



'C/E approaches 
one 



I E/C < ' 75 ' 
treatment 

favored 



large & negative 
control favored 



1 1 . S E < S c & < X E » 75 i 'E/Ci 1,0 ' smal I & positive 

nodi f ference 



greater 
than one 



control group 
favored 



'C/E approaches 
zero 



'E/C < • 75 » 
treatment 

favored 



largos & positive 
treatment favored 



II. S E >S C & Xc > X E I 



E /q close, to one, 
no dl f ference; 
"significantly" 
larger than one , , 
control favored 



smalls negative 
no differences^ 

large & negative 
control favored 



IV. S c > S c & Xc < X E 



.75 <_ l E/c <^ 1.0, 
no dl f ference; 
"significantly" 
larger than one 
control favored 



small & positive 
no dl f ference 

large & positive 
treatment favored 
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a number of experimental situations In addition to studying this 
distribution of the index (see Appendix B, derivation of the large 
sample normal approximation for Ie/c)*. F° r purposes of th.is* 
discuss;f6?Tw^Wi W adopt the convention that close to zero implies, 
0 <^ \£/q <_ .35; cloise to one will mean, .75 £ Ig/c <. 1.0; and Ig/c 
between .35 and .75 will be considered an "average" effect. 




'RefcalVTSlass 1 second example: _ x 

T 4 v . • • ■ 

v Method A Method B Control 

Means . 50- 50 48 

Standard Deviations -10 1 4 

- Applying the Glass estimator to these results, recalling that 
the standardization metric is Sq = 4, ES is equal to .5 for both 
methods, suggesting both conditions have equal effectiveness (recall 
that this is effectiveness relative to the 1 evel . of achievement 
and not effectiveness relative to the reduction of variance), in 
spite of heterogeneous variances. If one were to use the ES method 
to estimate the relative effectiveness between Methods A and B, 
regardless of the standard deviation (i.e., S/\,»= 10 or Sb -■!)» ES 
would equal zero suggesting no difference. A close inspection of 
the results of this experiment, once again fr^nf ! a ML viewpoint, 
suggests that Method B is more effective than Mtethod' A. 

-12- 



a number of experimental situations i.n addition to studying the 
distribution of the index (see Appendix B, derivation of the large 
sample normal approximation for Ie/c).. F° r purposes of th.is* 
discussfS^Tw^Pwi Kl adopt the convention that close to zero implies, 
0 <^ Ig/c <. .35; cloise to one will mean, .75 <^ Ig/c <. 1.0; and Ig/c 
between .35 and .75 will be considered an "average" effect. 




^Retall^Glass 1 second example: _ x 

v Method A Method B Control 

Means . 50. 50 48 

Standard Deviations "10 1 4 

- Applying the Glass estimator to these results, recalling that 
the standardization metric is Sq = 4, ES is equal to .5 for both 
methods, suggesting both conditions have equal effectiveness (recall 
that this is effectiveness relative to the level of achievement 
and not effectiveness relative to the reduction of variance), in 
spite of heterogeneous variances. If one were to use the ES method 
to estimate the relative effectiveness between Methods A and B, 
regardless of the standard deviation (i.e., S^- 10 or Sb - 1)> ES 
would equal zero suggesting no difference. A close inspection of 
the results of this experiment, once again f rprrf ! a ML viewpoint, 
suggests that Method B is more effective than Mtethod' A. 



PROPERTIES OF THE EFFECT VARIABILITY INDEX 



It <:an be sho^/n that Ig/c can be expressed as a linear function 
of ES (see Appendi\ A for derivation); i.e., 

■* 

I E/C j CV E /CV C = S E /S C - (CV E ).(ES) . (.3) ' 

Thus when ES = 0 (equivalent to \ = Tq) 9 If/c = $E/ S C> is \^ 
simply a function of the ratio of standard deviations* Consequently, 
the meaning, of Ig/c is apparent in the presence of equal treatment 
means and potential ly heterogeneous variances. , . 

In the reviewers/notes concerning an earlier version of this 
paper (Gljss, Shepherd and Smith, 1983), it was noted that in the 
presence of equal Standard deviations, Sg = Sq, the I^/c index. is 
not invariant under a linear transformation' 1 . The /foil owing example 
will, serve to illustrate this condition." Assume Sg = Sp, = 5 and "Xc 
= 45 and \ = 55. Then = .82 and ES = 10. If the means were 

transformed by adding 10 to each, then Ig/ft = -.85, whereas ES ' 
remains constant, i.e., ES = 10. If the sample sizes for both the' 
control and treatment groups were, ng = n^ 4,50, and applying t be 
large sample normal approximation results discussed in Appendix/ B, 
both the initial index and the transformed index yield a nonsig- 
nificant difference between the treatment and control^ The larger 
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the additive constant the greater the difference between the 
original Ie/c and the transformed estimate of effect variability. 



We must ask how meaningful this problem is in a ML context. 
Obviously, for experiments using the same measuring instrument 
(i.e. tests), there is no such transformation, and so there is no 
problem. If the instruments used are similar (in metrical properties) 
then the problem will be small, . If, however, the instruments are 
considerably different, then it would be possible to attain differences 
in I[:/c that looked important, but were due only to the particular 
choice of instrument. It -is the author's contention that. the 
problem here is in the use of inappropriate measures and not in the 
index itself. One way to circumvent such problems would be to 
ensure that all instruments used in, a series of ML experiments 
could be located on a common scale (one could use a Rasch linking 
procedure, outlined, in Best Test Design , Wright and Stone 1979); it 
is hard to imagine an ML experiement in which this would not be good 
advice in any case. If one is comparing across experimental studies, 
however, and the instruments used in each differ markedly in their 
level of difficulty, then this problem remains.. Such problem^/^^ 
these are not uncommon in indices and yet have practical- application. 
Ror example, Hedges (1981) has shown that ES itself^is biased even 
under homogeneous variances; however, thi^-xan be controlled and 
eve|i ignored, so* long as one is aware of the limitation in application; 



.In the presence of both unequal means and standard deviations, 

the situation improves. Assume =5 and Sq = 10, \ = 55 and 

Xc = 45.- Then Ig/c = .41 whereas ES = 10. Transforming both means 

by adding 10 to each yields Ig/c * a S'Ught increase of .01. 

Finally in the presence of equal means and unequal standard 

deviations, I^/c remains invariant, as illustrated by .Equation 3. 
In general if X 1 = aX + b then 

> .I'E/C = (SE/S^-KaXc + b)/(aX E + b)] (4) 

'^f ■ _ • : / ' 

and iE/c^I'E/e^" a V only if, b/a < Tc and b/a < The scaling 

parameter^ a, does not change the ratio of standard deviations. 



Contingent upon the previous discussion, it appears appropriate 
to recommend that Ie/Ci the effect variability index, be used as a 
tool for assessing experimental outcomes within a given research 
study -as opposed to between studies, relative to meta analysis 
concerns. However, if tests used in a variety of experimental 
situations are equivalent in their level of difficlty, the Ig/c 
estimator is a suitable tool to employ.. 
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Applications to real data 

The reader is reminded that the consideration of Ig/c has been 
stimulated by experiments associated within the ML context, where 
level and variability of achievement are both of concern as measures 
of effective ML instruction, a problem not adequately accounted for 
by the Glass estimator. The data examined in Table 2 are from a 
study undertaken by Block (1972) in which the criterion level of - 
mastery was selectively varied across four treatment groups (95, 85, 
75 and 65 percent mastery. levels) and the results compared to a 
control, non-mastery group! Block noted "that as the performance 
level attained increased, average achievement test scores rose and 
the dispersion fell. The 85 and 95 percent mastery treatments not 
only helped students to attain significantly higher average scores 
than the non-mastery treatment, but the treatments also helped 
homogenize student achievement around these scores." 

I 

For the 75 and 65 percent mastery learning groups, the ES 
estimator is approximately zero; whereas I75/C ' «?3 and 155/0 = *69 
and "average" effect according to the convention adopted earlier. 
For the 95 and 85 percent mastery groups, ES = .64 and ES = .46 
respectively; whereas, I95/C = *31 and I85/C = - 39 > effects strongly/ 
favoring the two ML groups. These differences are due primarily to 
the Impact -of the ratios of the standard deviations. favoring the 
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TABLE 2. Comparisons Between lg C and ES for Block Data 



Average 

Test Scores Standard ' _ 

Treatment Group (Percent Correct) Deviation < s m/ s NM> x ^ x fW /x M ) s ! E/C ES 



Mastery 

95*percent 64.9 9.09* .41 .77 .31 .64 
(N =$1) 

85 percent 60.7 10.49* .47 .83 .39 .46 
(N = 14) 

75 percent 50.8 11.79*. .53 1.00 .53 .01 
(N - 14) 

65 percent 49.0 15.49 .69 1.00 .69 -.07 

(N = 12) . 

Non-mastery 50.5 22.40 
(N 3 25) 



*Slgn If leant difference (p < .05) between treatment group standard deviation compared 
to control group. 
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experimental groups. Of the four comparisons permitted by the, F.S / 
method, the 95, 85 and 75 percent mastery treatment groups 1 standard 
deviations are significantly different from the non-mastery group 
standard deviations, (The ;F-test for the ratio of two variances 
was employed.) For the 75 percent mastery group comparison, 
ES = .01, suggesting no difference between the treatment and control 
group; whereas l£/ C = .53, implying a difference between the groups 
compared in the presence of. J^eterogeneous variances. 

Table 3 present^ the results of the matrix of all possible 
comparisons among the treatment groups along with the confidence 
intervals for each calculated Ie/C ( see Appendix B for confidence 
interval formula). For example, reading down the 95 percent column, 
in which the 95 percent mastery group is compared to all other 
groups, each confidence interval containing 1.0 indicates a non- 
significant difference between the groups compared. The ES estimato 
does not permit comparisons among the treatment groups, whereas the 
Ie/C estimator, together with the confidence intervals, implies 
that no differences exist among the 9b, 85, and 75 percent groups; 
between the 75 and 65 percent groups'; and between the 65 percent 
and' non-mastery group. Al 1 otherylif ferences between the treatment! 
groups and the non-mastery group/are significant. 4 Once again 
although the. ES = .01 suggests no difference for the* 75 vs non- 
mastery group comparison, th</l£/c estimator does indicate a 
difference between these groups. 



\ 

TABLE 3. Matrix of Comparisons of I E /q for Block Data 



j Treatment Groups . ES Estimator- 

95 85 75 65 

95 .64* 

85 81 

I " 

Treatment (.20, 1.42) 1 .46* 

Groups 

75 .60 * .74 

(.14, 1.06) (.21, 1.27) .01* 

65 .44 .55 .73 . 

(.09, .79) (.T3, .97) (.17, 1.29) -.07 

Nonmastery .32 .39 , .52 .71 

(.09, .55) (.13, .65) (.17, .87) (.20, 1.22) 

*Slgn If leant difference (p < .05) between treatment group standard deviation compared 
to control group. « 

1. See Note. All confidence Intervals are computed at the 99 percent level. 



The second set of comparisons, presented in Table 4, come from 
a number of mastery learning experiments reported by Bloom (1976), 
The data have been rank ordered relative to the ES estimator and 
separated into two groups, heterogeneous and homogeneous variances. 
All confidence intervals for Ie/c not containing 1.0 imply a signif- 
Tcaht difference favoring the ML group. Of the 8 comparisons within 
the heterogeneous group, all Ie/c estimates are significantly 
different from 1.0. Of the 11 comparisons' for the homogeneous group 
only 2 l£/c estimates are significant, favoring the ML group. It 
appears that in the presence of heterogeneous variances, the Ig/c 
estimator parallels the ES index suggesting an effect favoring the 
ML group; that is, not only is effect size large relative to level 
of achievement (ES estimator) but., also large relative to the 
reducti'on of variability Ue/C estimator) (This condition is true 
for all cases with one exception , noted in the Block data in Table 2, 
for ES = .01 and I75/NM = .52.). However, the "magnitude" of the 
effect implied ,by the.- Ie/c estimates and the corresponding confi- . 
dence intervals are not as strong as those implied by the ES index 
based upon the previously suggested -guidel i nes. -In the presence of 
homogeneous variances, the results are reversed. For the 8 ES 
estimators greater than the absolute value of .49, only one" Ie/c 
index is significant, compared to the largeeffect sizes reported. 
Consequently, in the presence of homogeneous variances, cal though 
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TABLE 4^ Comparisons Belueen l E /c and ES for Bloom Data 



' X E S E N E 1 Xc S c Mc l E/c Le/C Confidence ES 
~ "~ — 1 ' Interval .Estimator 



Heterogeneous Variances 



i a nn 


18.40 


1985 


55 ^0 

J J . JU 


19.10 


1271 


.72 


(.67. 


.77)* 


.98 


D U a / U 


10.10 


15 


46. 50 


1 5.00 


93 


.52 


(.25, 


.79)* . 


.95 




17.30 


1723 


49.50 


22.30 


1310 


'.55 


(.51 , 


.59)* 


.89 , 


71.40 


21.40 


1895 


52.80 


21.10 


1410 


.75 


(.69, 


.80)* 


.88 


78.20 


4.50 


1 13 


72.40 


6.70 


113 


.62 


(.47, 


.77)* 


.87 


73.50 


19.00 


1806 


54.30 


22.30 


1 104 


.63 . 


.' (.58, 


'.68)* 


.86 


3.52 


0.50 


98 


2.51 


1.40 


75 


'.26 


(.17, 


.34)* 


.72 


90.30 


. 5.20- 


20 


85.90 


10.10 


21 


..49 


(.20,. 


.76)* 


.44- 


jmoqeneous 


Variances' 
















30.30 


3.80' 


30 


26.10 


4.70 


21 


.70 


(.33,' 


1.07) 


.89 


64.40 


16.30 


19 


52.80 


13.20 


24 


1.01 


(.41 , 


1.61 ) 


.88 


74.60 


12.40 


33 


64.40 


12.60 


20 


.85 


(.39, 


1.29) 


.81' 


68.50 


13.90 


26 


56.90 


16.00 


33 


.72 . 


(.36, 


1.08) 


.72 


120.30 


23.90 


17 


99.00 


35.40 


17 


.56 


(.18, 


.94)* • 


.60 


7 3.10 


12.10 


168 


66,80 


J 1.90 


92 


.93 


(.70, 


1.16) 


.53 


3.46 


0.59 


24 


' 3.07 


0.79 


22 


.67 


(.29, 


1.05) 


.49 


77.10 


12.00 


20 


71.40 


J 14.70 


17 


.76 , 


(.29, 


1.23) 


.39 


61.30 


8.10 . 


9 


■ 56.10 


'Tj4.10 


- " 9 


.52 


(.06, 


.98)* 


.37 


13.50 


4.50 " 


,34 


15.90 


5.60 


37 


.94 


(.49, 


1.39) t 


, -.43 


59.00 


14.80 


31 


67.40 


17.00. 


30 


" .99 


(.50, 


1.48)* 


-.49 



*Slgnlf leant difference between the ML and control group. All confidence Intervals 
•are calculated at tha 99 r percent level. 



the ES estimator is large relative tothe level of ."achievement 
differences, the reduction in the variability is minimal. 

Table 5 present^Che results of Table 4, in a different prespec- 
tive, emphasizing the importance of-examining the component ratios 
of means and standard deviations, in order to clarify the results of 
the preceding discussion. The format of Table 5 parallels that of 
Table 4. The first apparent trend is that the component ratios of 
the means and standard deviations for the homogeneous variance group 
of comparisons exceed those of the heterogeneous variance group, 
indicating that the means and standard deviations of the ML and 
icontrol groups are approximately equal numerically in the homogeneous 
group. Of the two significant I^/q indices in the homogeneous group, — 
one apparent explanation for their significance is implied by the two 
smallest standard deviation ratios, .68 and .57 associated with nearly 
equivalent mean ratios of .82 and .92 respectively. That is,, in the 
case of Sg/Sc = .57, the ML groups 1 standard deviation is only 57 
percent of the control ■ group's standard deviation; and in the 
presence of a mean ratio Yq/Te' - results in significance. A 

similar occurrence is exhibited in the heterogeneous group of 
comparisons for S^/Sq equal to .67 and .51 relative to Yq/Ye equal 
to .93 and .95 respectively. Consequently, the interactions of the 
two component ratios comprising: \£/q do contribute to the inter- 
pretation and understanding of l£/o Simulation studies are 
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TABLE 5. Comparisons of the Component Ratios of the Ig/e Index 



S E/ S C Xq/Xe Ie/C ES Estimator 



Heterogeneous Variances 



.96 


.75 


.72 


' .67 


.77 


.52 


.78 


.71 


.55 


KOI 


.74 


.75 


.67 


.93 . 


.62 


.85 


.74 


.63 


.3f 


.71 


.26 


.5' 


.95 


.49 


Homogeneous 


Variances 




.81 


.86 


.70 


1.23 


.82 


1.01 


.98 


.86 


.85 


.87 


.83 


.72 


.68 


.82 


.56 


1.02 


.91 


.93 


.75 


.89 


.67 


.82 


.93 


. .76 


.57 


.92 


.52 


.80 


1.18 


.94- 


.87 


1.14 


.99 



.98 
.95 
.89 
.88 
.87 
.86 
.72 
.44 



.89/ 

.88 
^.81 

.72 

.60 

.53 

.49 

.39 

.37 
-.43 
-.49 • 
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directed at investigating the effects of sample size relative to 
the sampling variance' of Ijr/c are reported in Appendix B. 

SUMMARY 

The index proposed in this paper is advanced as an additional 
measure of treatment effect, specifically applied to experimental 
programs where both level and variability of the variable under 
study is of dual concern. The proposed effect variability esti- 
mator, compared to the ES estimator and presently used to measure 
the effect of ML programs: 

1. utilizes the means and standard deviations of all treatment 

conditions to be compared; / 

/ • • 

1. does not go to zero in the presence of equal means and 

hi^eroge^ hoiffogenew 

provides for enhanced interpretation \and understanding via a 
straightforward examination of the "component ratios and 
kheir relative impact; 

4. facilitates "the rank ordering of treatment effects within' 

.■V' \ 

experimental studies among all treatment groups;. 

5. emphasizes the dual nature of- "effect size" relative to 
both level of achievement and reduction of variability 
within the context of ML experiments. 
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The importance associated with both level and variability of 
achievement, especially relevant to ML experiments, demands the 
development of techniques to capture and illuminte these conditions. 
This paper is an attempt to resolve a preplexing problem. At 
present work is underway, directed at investigating the distri- 
butional properties of Ie/c and comparing , the power of Ig/c rela- 
tive to ES under the conditions discussed. 
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APPENDIX A 
Derivation of Ig/c as a Linear Function of ES 



r 
I 
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ES - 



sc 



r 



ES 



S C 



es = y E . 



ES = _ 



1 - X C /Xe 

_ x c " 

I E " 



se 



sc 



se 
xe 



ES = 



(ES) 



xe 
sc 



Se xc / s E 
_Sc Sc x"e 

/ 

(CV E )/ / (CV C ) 



Ie/c = cv e/cv c = - (cveWes) 
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APPENDIX. B 
Derivation of the Large Sample Normal Approximation for I e/c 



Er|c 3? 



LARGE SAMPLE NORMAL APPROXIMATION 



The derivation of the large sample normal approximation for 
lE/C appeals to the delta (£) method discussed In Rao (1973): 
Let T n be a k-d1mens1onal statistic (Tj n| . . .T|< n ) such that the 
asymptotic distribution of ^fiT(Ti n - Q]) 9 1 - l...k, is 
k-variate normal with mean zero and dispersion matrix 2= (Glj). 
Let gi ...g q be q functions of k-varfates and each q\ be. totally 
differentiable. Then the a.d. of VrT C91 (Tin , •T| cn )-91 ( e l ,•• - 0 k)]» 
1 = 1, ...,q, is q-variate normal with zero means and dispersion 
matrix G2G\ where G = (Sgi/fQj). For Ie/C, 1et 




and 



0 = 



hi 2 

u 2 

V' 1 / 



(1) 



where Tf^N^i^), and assume that the asympototic distribution of 
VfT (T - 0) i s N(0> 5, ) where 2 1s defined as 



21^ 


0 


0 


0 . 


2T 2 4 


0 . 


0 




T 2 2 


n 


0 


0 




(2) 
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Define the function g to be g » V T]/T2 (T3/T4), then the £ -method 
method yields 

^Cg(X)-9(o)]-N(o,A2A') (3) 

where A = (a i j ) and a ^ j = ig/^Tj evaluated at T » 0 . Taking the 
partial derivatives of g(T) and calculating AZA' yields 



(4) 



Evaluating .A 2 A' at T = J9 and recal ling that (Oi = T1 2 , 0 2 » T2 2 , 
O3 = U2 and O4 = ui) yields 

(T. 2 = A£A' = (CTi?^ 2 )- (u 2 /ui) 2 [1 +CT 2 2/u 2 2 t G\2/\i\ z l (5) 

The asymptotic distribution of Ig/c is 

>Tn" [I E/C - (Gi/u 1 )/(CT2/u2):i~N(0,A2A') (6) 

and the corresponding large sample normal approximation is 

\ 

I E /C^NC((T 1 /u 1 )/(G2/u2), ASA']. - (7) 

Substituting the respective sample statistics results in a test 
statistic of no difference between the treatment groups of 
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and a correspond! ncj confidonco Interval of 



. »E/C- (^./nHT) < CV E /CV C < l£/C + ? -«/2 ( ff -/^n). (9) 

* "\ 

i 

For unequal sample sizes 0".^ becomes . \ 

cr.2 - C( cr 1 /(r ? ).(u 2 /u 1 )]2^.^~^ + H (<r 2 /u 2 ) 2 + h Pi/ui)*). (l i n) 

where N = + ng, and I^/c^N C (CTj ATg) / (^2/^1 ) » 

ACCURACY OF THE LARGE. SAMPLE APPROXIMATION 

The statistical manipulations above result in a large sample 
approximation to the distribution of Ig/o In order to check the 
usefulness of this approximation for small samples, a simulation 
study was conducted. The subroutine GGNML of the International 
Mathematical and Statistical Libraries (1977) was used to generate 
standard normal N^ev^ate^^ Bl, gives the represenative values 

of l£/c used the* s'imul ations. These were chosen to represent a 
range of values that would be considered likely in experimental 
studies such as the mastery learning studies under consideration 
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tffi'ere. Caution. is recommended, however, in making interpretations 
beyond the range reported. 



In each simulation, th$ experimental and control group sample 
sizes were "chtfsen to be equal, that, is n = n^ = n^. The normal 
deviates were. generated as two independent vectors of length n, 
one fo N r the experimental group and one .for the control group. "The 
sample means and standard deviations for each wer^e then found, 
allowing the computation of Ie/O' The me ,?P s and variances of the 
simulated Ie/C* s » for the 14 types of simulation chosen, are given 
in Tables R2, B3 and B4. Also shown .are the proportions of confi- 
dence intervals which contained the true value of Ijryc with certain 
nominal significance levels.- These proportions were found by 
noting, for each simulation, whether the true value of I tl /C was 
within the nominal confidence interval calculated using the ^sample 
value of Ie/C and the sample .value of (T. (as given in equation 9);. 
these . insj^ani^e-s^^ for each of the nominal signifi- . 

cance levels and then divided by the total number of simulations^ 
give the result. The proximity of thse proportions to the, nominal 
level is' : an indication of the accuracy of the large sample approxi- 
mation. 

The first series of simulations a re^ y reported in Table R2; this 
series contrasts the performance of/the approximation as the 



/ 
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experimental standard deviation, infl ates. and deflates, while the 
other parameters remain constant. Consider first the column of 
means; this column revealed a tendency to overestimate .Ie/c» T^ e 
bias declines from about 6 percent for the case n = ID, to about 
2.5 percent for n = 20, down to about 1 percent for n = 50. Thi\s 
bias can- be traced to the fundamental assymmetry in the distribution 
of .the standard errors; it is of noteworthy size fo^ the simulations 
at n =10, and such cases should thereby 1 be treated with caution, 
but its performance for the large sample sizes is quite within 
the bounds suitable for practical application. The story told by 
the proportion of confidence intervals containing I^/c is rather 
similar. Here, all values found match the nominal ones quite well 
except for the n = 10 cases. There is a small but marked tendency 
for the empirical proportion to be below the nominal ones. This 
may be due to bias of the estimator , V tendency for the variance 
estimator to underestimate the twice variation, or a combination 
of the two. This issue cannot be fully clarified until an unbiased 
estimator is found. , 

The second series of simulations is reported in Table B3; this 
series contrasts the performance of the approximation as the experi- 
mental mean shifts, and the other parameters remain constant. i The 
results give the same, picture: the approximation's doing quite 

, • , / \ ■ ■ 

well for the case n- = 20 and upwards, but^is poor for n =.10. The 
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third series of simulations is reported in, Table R4; here the 
experimental means and standard deviations are simultaneously . : 
manipulated, but the experimental xoeffi ci ent of variation and the 
control group parameters are kept constant. In this series also, 
^the, approximation holds up well for n = 20 and upward, hut is poor' 
for n = 10. 

These results indicate that (subject to the previous remark on 
caution in extrapolating simulation results) the approximation is 
useful when there are 20, or more cases in each of the experimental 
/and control groups, but that it should not be used for smaller 
sample groups. Work is continuing on finding a correction for the 
bias in the estimator I.e/o ' 0 , ' 
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Simulation 



1 
2 
3 
4 
5 
6 
7 
8 
9 

in 
11 

12 

.li 



l E/C 



TABLE Bl. Representative Values of 



'C 



14, 



1.00 
1.25 
1.50 
0.75 / 
0.50/ 
0.25 
0.-9091 
f>{8333 
A. WW 
1.25 
1.00 
1.00 
1.00 
1.00 — 



10.0 


1.0 / 


.10.0 


10.0 


• 1./ 


10.0 


10.0" 


I/O 


10,0 


10.0 


• /1.0 \ 


10.0 


10.0 


'/1.0 


10,0 


10.0 


/ 1.0 


10.0 


10.0 / 


/ 1.0 


11.0 


. 10.0/ 


1.0 . 


12.0 


10.0 


1.0 


9.4 


10.0 


1.0 


8'. 6 


10.0 


1.0 


14.0 


10.0 


-1.0 


12.0 


10.0 . 


1.0 


8.0 


- 10-. 0 - 


-2 -LJ) 


6.0 



1.00 
1.25 
1.50 
0.75 
0.50 
0.25 
1.00 
1.00 
1.00 
1.00 
1.40 
1.20 
0.80 
,0.60] 
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Simulation 



1 
2 
3 
4 
5 
6 
7. 
8 
9 

in 
n 

12 



l E/C 



TABLE 81. Representative Values of 



14. 



1.00 
1.25 
1.50 
0.75 /' 
0.50/ 
0.25 
0.4091 
0/8333 
/t.Ul'l 
1.25 
1.00 
1.00 
1.00 
1.00 — 



'C 



10.0 


1.0 / 


-10.0 


10.0 


• w 


10.0 


10.0" 


1..0 


10,0 


10.0 


• /1'.0 . 


10.0 


10.0 


'/1.0 ' 


10,0 


10.0 


/ 1.0 


10.0 


10.0 / 


/ 1.0 


11.0 


. 10.0/ 


1.0 . 


12.0 


10.0 


1.0 


9.4 


10.0 


1.0 


8'. 6 


10.0 


.1.0 


14.0 


10.0 


■ 1.0 


12.0 


10.0 . 


1.0 


8.0 


- 10-. 0 - 


_LJ) 


6.0 



E 



1.00 
1.25 
1.50 
0.75 
0.50 
0.25 
1.00 
1.00 
1.00 
1.00 
1.40 
1.20 
0.80) 
,0.601' 
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'TABLE B3. . Small Sample Accuracy of Qo'nfidence Intervals for lc/r' 
Equal Standard Devi ations^and Differing Means 



Sample Size- Mean 
n/= n E . n c . I £/c 



Variance / 

h/c 



Proportion of confidence intervals 
containing withrj.ominal 

significance >evel 



.60 



.70 



.80 



.95 



.99 







Simulation 


7 




O.-969I 






/ 


10 


.9553 


.1202 


.576 


sm. 


.763 


.865. 


.910 


.952 


20 


.9385 


.0482 


.599 


X^.695 


.785 


.890 


.947- , 


.980 


30 


.9193 


.0302 


.6,02^ 


.688 


.789 


.898 


.94.2 


.978 


/in 


Q?5n 




/60 8 


707 


.807 


.904 


.952 


.984 


50 


• v .9139 


.0184^/ 


.583 


.679 


.790 


.885 


.941 


.984 






Simul at ion 


8 


F/C = 


0.8333 








10 


.8958 X 


.1047 


.571 


.680 


.784 


.873 


.918 


.962 


• 20 


J35154 


.0421 


.582 


.686 


.782 


.880 


.934. 


.969 ' 


30 


X8452 


.0271 


.581 


.686 


1 .799 


.886 


.929 


, .971 


40 


X -8467 


.0204 


.574 


.674 


.776, 


.889- 


.941 


' .978 


50 / 


.8383 


.0150 


.606 


.700 


.798 


.886 


.938, 


.985 






Simul ation 


9 . 


! E/C = 


1.1111 








10 


1.1865 


.1849 


.563 


.667 


.775 


.870 


.914 


.952 


20 


1.1483 


.0792 


.596 


.695 


.794 


.887 


.936 


.978 


30 


1.1293 


.0479 


.579 


.690 


.785 


• .886 


.936 


.979 


40 


l.UGT: 


.0337 


.603 


.702 


.802 


.894 


.944 


.983 


50 


1.1216 


.0290 '• 


.586 


.687 


.790 


.881 


.929 


.980 






Simul ati on 


10 


r E/C : 










10 


1.3273 


.2561 


.558 


.665.. 


.771 


.869 


.' .920 \ 


.962 


20 . 


1.2830 


.0986 


.581 


.689 


.791 


.887., 


.936 ' 


.976 


30 


• 1.2704 


.0609 


.588 


.676 


.786 


.893 


.940 


.976 


40 


1.2771 


.0469 


.580 


.676 


.783 


.891 


.945 


.985 


50 


1.2642 


.0355 


.565 


.666 


.778 


.880 


.941 


.981 



NOTE: Each of the simul ati ons . i s based on 2000 cases. 
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TABLE B4. Small Sample Accuracy of Confidence Intervals for I 
/ Differing Means and Standard Deviations 



Proportion of confidence intervals 
Sample Size Mean Variance containing Ir /r with nominal 



n = n^ = n^ 1^ ,q *E/C - significance level 





V, 1 / V, 
w v w f w 




.60 


.70 


.80 


.90 


.95 


.99 


- 




Simulation 11 


! E/C = 


uo 








10 


1 .0620 


.1593 


.552 


.645 ' 


.745 


.846 


.899 


.948 


20 t 


1.0224 


.0649 


.579 


.678 


.782 


.872 


.917 


.972 


30 ' ". 


1.0097 


.0364 


.5.96 


.693 


.792 


.893 


.944 


.979 


40 


1 .0159 


.0285 


.600 


.694 


.791 


.900 . 


.946 


.986 


50 


1.0066 


.0212 


.595 


.694 


.793 


.897 


•945 


.981 






Simulation 12 


! E/C = 


1.0 








10 


1 .0850 


.1712 


.549 


.652 


.7 59 ' 


.868 


.916 


.955 


20 


1.0372 *• 


.0612 


.585 


.68"4 


.789 


.890 


.941 


.975 


30 


1.0180 


.0407 


' .561 


.664 


.775 


.876 


.930 


.979 


40 


1.0187 


.0279 


.591 


.692. 


.798 


.898 


.945 


.982 


50 


1.0081 


.0227 


.584 


.693 


.795 


.892 


.943 


.985 


r 




Simulation 13 


! E/C = 


1.0 








10 


1.0732 


.1649 


.572 


.673 


..774 


.875 


.918 


.960 


20 


1.0328 


.0653 


.576 


.678 


.779 


.876 


.922 


.966 


30 


1.0208 


.0382 


.601 


.693. 


.795 


.891 


.939 


.979 


40 


1.0103 


.0285 


.596 


.689 


.791 


.885 


.933 


.980 


50 


1.0138 


.0227 


.595 


.695 


.796 


.8.90 


.941 


.980 






Simulation 14 


l f.fC = 


i.o 








10 


1.0676 


.1579 


.566 


.666 


.776 


.880 


.921 


.959 


20 


1.0252 


.0615 


.585 


.689 


.784 


.880 


.931 


.968 


30 


1.0158 


.0382 


.590 


.694 


.801 


.887 


.939 


.977 


40 


1.0131 


.0280 


.598 


.697 . 


.797 


.896 


.938 


.980 


50 


1.0131 


.0217 


.585 


.685 


.802 


.902 


.948 


.983 



NOTE: Each of the simulations is based on 2000 cases. 
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